-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix attnum mismatch in pg_statistic after drop column and database restore #90
Conversation
NOTE: If one wonders about possible performance impact of the delta, here is some execution time measurement done. On a db with 2000 tables (with 120 columns each), the backup time with statistics is following: with the fix:
without the fix:
which doesn't look dramatic. The code for creation test tables is following:
|
This comment was marked as resolved.
This comment was marked as resolved.
bender build |
restore/restore.go
Outdated
@@ -415,6 +415,10 @@ func editStatementsRedirectSchema(statements []toc.StatementWithType, redirectSc | |||
permissionsRE := regexp.MustCompile(fmt.Sprintf(`(?m)(^(?:REVOKE|GRANT) .+ ON .+?) (%s)((\..+)? (?:FROM|TO) .+)`, schemaMatch)) | |||
// This expression matches an ATTACH PARTITION statement and captures both the parent and child schema names | |||
attachRE := regexp.MustCompile(fmt.Sprintf(`(ALTER TABLE(?: ONLY)?) (%[1]s)(\..+ ATTACH PARTITION) (%[1]s)(\..+)`, schemaMatch)) | |||
// This expression matches a '<schema>.<table>'::regclass::oid expression | |||
regclassOidRE := regexp.MustCompile(fmt.Sprintf(`'(%s)((\.[^']+)'\:\:regclass\:\:oid)`, schemaMatch)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this regex does not catch table name with single quote
create schema qwe;
create schema rty;
CREATE TABLE qwe."fo'o"(i int);
INSERT INTO qwe."fo'o" VALUES (1), (2), (3);
gpbackup --backup-dir ~/gpbackup --debug --dbname gpadmin --with-stats --single-backup-dir --include-schema qwe
gprestore --timestamp 20240809084649 --debug --backup-dir ~/gpbackup --on-error-continue --redirect-db gpadmin --redirect-schema rty --include-schema qwe --with-stats
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Restore Command: [gprestore --timestamp 20240809084649 --debug --backup-dir /home/gpadmin/gpbackup --on-error-continue --redirect-db gpadmin --redirect-schema rty --include-schema qwe --with-stats]
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Restore Key = 20240809084649
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-gpbackup version = 1.30.5_arenadata15+dev.17.g597f3e4b
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-gprestore version = 1.30.5_arenadata15+dev.17.g597f3e4b
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Greenplum Database Version = 6.27.1_arenadata56+dev.36.g6ad72daf603 build dev
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Gathering information on backup directories
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Verifying backup directories exist
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Metadata will be restored from /home/gpadmin/gpbackup/backups/20240809/20240809084649/gpbackup_20240809084649_metadata.sql
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Restoring pre-data metadata
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: CREATE FUNCTION rty.f() RETURNS void AS
$$begin
DELETE FROM pg_statistic WHERE false;
end;$$
LANGUAGE plpgsql NO SQL; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 10% (1/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: ALTER FUNCTION rty.f() OWNER TO gpadmin; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 30% (2/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: CREATE TABLE rty."fo'o" (
i integer
) DISTRIBUTED BY (i); on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 50% (3/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: ALTER TABLE rty."fo'o" OWNER TO gpadmin; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 60% (4/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: CREATE VIEW rty.my AS SELECT count(*) AS count
FROM pg_statistic; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 80% (5/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: ALTER VIEW rty.my OWNER TO gpadmin; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Pre-data objects restored: 100% (6/6)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Pre-data metadata restore complete
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Verifying backup file count
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: SET client_encoding = 'UTF8'; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: SET client_encoding = 'UTF8'; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Restoring data for 1 tables from backup with timestamp: 20240809084649
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: SET client_encoding = 'UTF8'; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: SET client_encoding = 'UTF8'; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Reading from /home/gpadmin/gpbackup/backups/20240809/20240809084649/gpbackup_<SEGID>_20240809084649_16387.gz
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing "COPY rty."fo'o"(i) FROM PROGRAM 'cat /home/gpadmin/gpbackup/backups/20240809/20240809084649/gpbackup_<SEGID>_20240809084649_16387.gz | gzip -d -c' WITH CSV DELIMITER ',' ON SEGMENT;" on master
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Restored data to table rty."fo'o" from file (table 1 of 1)
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Data restore complete
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Restoring post-data metadata
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Post-data metadata restore complete
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Restoring query planner statistics from /home/gpadmin/gpbackup/backups/20240809/20240809084649/gpbackup_20240809084649_statistics.sql
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: UPDATE pg_class
SET
relpages = 2::int,
reltuples = 3.000000::real
WHERE oid = 'qwe."fo''o"'::regclass::oid; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: DELETE FROM pg_statistic WHERE (starelid, staattnum) IN
(SELECT attrelid, attnum FROM pg_attribute WHERE attrelid = 'qwe."fo''o"'::regclass::oid AND attname = 'i'); on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[DEBUG]:-Executing statement: INSERT INTO pg_statistic SELECT
attrelid,
attnum,
false::boolean,
0.000000::real,
4::integer,
-1.000000::real,
2::smallint,
3::smallint,
0::smallint,
0::smallint,
0::smallint,
97::oid,
97::oid,
0::oid,
0::oid,
0::oid,
NULL::real[],
'{"-0.5"}'::real[],
NULL::real[],
NULL::real[],
NULL::real[],
array_in('{"1","2","3"}', 'int4'::regtype::oid, -1),
NULL,
NULL,
NULL,
NULL
FROM pg_attribute WHERE attrelid = 'qwe."fo''o"'::regclass::oid AND attname = 'i'; on connection: 0
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Query planner statistics restore complete
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Found neither /usr/local/bin/gp_email_contacts.yaml nor /home/gpadmin/gp_email_contacts.yaml
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Email containing gprestore report /home/gpadmin/gpbackup/backups/20240809/20240809084649/gprestore_20240809084649_20240809084944_report will not be sent
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Beginning cleanup
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Cleanup complete
20240809:08:49:44 gprestore:gpadmin:gpdb6:060212-[INFO]:-Restore completed successfully
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Fix attnum mismatch in pg_statistic after drop column and database restore
Problem:
In case a column (or columns) was dropped from a table, after backup and restore
cycle, second backup may contain pg_statistic values with a type not matching
the data (for ex. "array_in('{"test"}', '"timestamp"'::regtype::oid, -1)").
Root cause:
Records from pg_statistic were backed up with their actual values, including
attnums for the columns. But during restore, all tables are restored from
scratch. So, if a column is dropped in the table before backup, after the
restore attnum values in the pg_attribute table differ from the attnum
values that were before the backup (if the dropped column is not the one with
the highest attnum). But the restored values of pg_statistic had old attnums.
So, pg_statistics was not consistent with pg_attribute, and one of the side
effects was that on next backup the type for pg_statistic value was taken from
other column (and some pg_statistic were just lost).
Fix:
On the restore phase the column name is used to obtain actual attnum value from
the restored catalog and this value is now used when restoring statistics. The
'AttNumber' field is removed from the AttributeStatistic struct to prevent
its future use.